Android是如何实现流量统计的?

网络管理流量统计

字数统计: 2.6k阅读时长: 12 min

 2020/04/01

使用Android手机时, 我们不仅可以看到当前系统的流量使用情况, 还可以查看每个应用消耗了多少流量, 借此我们可以发现有那些流氓APP在偷偷在背后消耗流量.那么, Android是具体如何实现流量统计的? 又是如何对每个应用的流量使用进行监控? 这篇文章我们就来看看Android流量统计的具体实现原理.

大致说来, Android从如何几个方面进行流量统计:

统计每个网口当前发送/接收的流量数据
监控每个应用(对应唯一的UID)所消耗的流量
支持对总的流量配额进行限制, 如达到一定的流量阈值后, 会对网络进行限制

而具体到每个应用(比如system应用, UID=1000), Android还支持对应用内的每个socket进行标记(tag), 用于区分每个应用(UID)内部具体使用了那些流量.后面, 我们会讲到如何通过标签来区分UID内部的流量.

下图是Android流量统计的原理框图: 为了实现流量统计, Android在Linux内核增加了一个netfilter模块: xt_qtaguid(源码可以在kernel/net/netfilter中找到), 用于统计当前系统所有流量, 该模块初始化时, 会初始化一个/proc/net/xt_qtaguid目录供用户空间的进程使用;NetworkStatsService系统服务就是周期性的读取该目录的数据来获取当前系统消耗的实时流量的;而如果要对某个特定的socket打上标签, 则需要通过JNI接口调用,然后发请求给netd将该socket标签信息通过接口/proc/net/xt_qtaguid/ctrl写入内核.

Android流量统计原理图

接下来就一起看下Android具体是如何进行流量统计的.

Android流量统计实现

Android有一个系统服务NetworkStatsService来负责流量统计管理. 在系统启动的时候会创建该服务, 对其进行初始化: 创建一个NetworkStatsService, 并返回给SystemServer, 服务内有一个线程用于数据统计业务的处理.


public static NetworkStatsService create(Context context,
            INetworkManagementService networkManager) {
    AlarmManager alarmManager = (AlarmManager) context.getSystemService(Context.ALARM_SERVICE);
    PowerManager powerManager = (PowerManager) context.getSystemService(Context.POWER_SERVICE);
    PowerManager.WakeLock wakeLock =
            powerManager.newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, TAG);

    NetworkStatsService service = new NetworkStatsService(context, networkManager, alarmManager,
            wakeLock, getDefaultClock(), TelephonyManager.getDefault(),
            new DefaultNetworkStatsSettings(context), new NetworkStatsObservers(),
            getDefaultSystemDir(), getDefaultBaseDir());

    HandlerThread handlerThread = new HandlerThread(TAG);
    Handler.Callback callback = new HandlerCallback(service);
    handlerThread.start();
    Handler handler = new Handler(handlerThread.getLooper(), callback);
    service.setHandler(handler, callback);
    return service;
}

等到SystemServer完成对系统服务的初始化后, 会调用NetworkStatsService.systemReady(), 告诉服务可以正常启动了, 启动时NetworkStatsService需要做如下几件事情:

创建四个流量统计的类型, 实际对应放在/data/system/netstats目录的四个类型的文件而已, 分别用于统计每个网口的消耗的流量(PREFIX_DEV), 视频通话以及热点分享所消耗的流量(PREFIX_XT), 每个用户所消耗的流量(PREFXI_UID)以及每个用户对应的每个标签所消耗的流量(PREFIX_UID_TAG)
更新每个流量统计数据写入的阈值: 即流量消耗达到某个阈值后, 需要将当前统计数据写入磁盘, 目前默认统一使用的是2MB；接着还要看下是否需要从早前版本中把老的流量统计数据迁移过来
注册并监听系统广播, 比如定时从系统拉取流量统计数据(ACTION_NETWORK_STATS_POLL), Android默认30分钟拉取一次; 系统用户增加与删除的广播;关机的广播等


public void systemReady() {
    mSystemReady = true;

    synchronized (mStatsLock) {
        // create data recorders along with historical rotators
        mDevRecorder = buildRecorder(PREFIX_DEV, mSettings.getDevConfig(), false);
        mXtRecorder = buildRecorder(PREFIX_XT, mSettings.getXtConfig(), false);
        mUidRecorder = buildRecorder(PREFIX_UID, mSettings.getUidConfig(), false);
        mUidTagRecorder = buildRecorder(PREFIX_UID_TAG, mSettings.getUidTagConfig(), true);

        updatePersistThresholdsLocked();

        // upgrade any legacy stats, migrating them to rotated files
        maybeUpgradeLegacyStatsLocked();

        // read historical network stats from disk, since policy service
        // might need them right away.
        mXtStatsCached = mXtRecorder.getOrLoadCompleteLocked();

        // bootstrap initial stats to prevent double-counting later
        bootstrapStatsLocked();
    }

    // watch for tethering changes
    final IntentFilter tetherFilter = new IntentFilter(ACTION_TETHER_STATE_CHANGED);
    mContext.registerReceiver(mTetherReceiver, tetherFilter, null, mHandler);

    // listen for periodic polling events
    final IntentFilter pollFilter = new IntentFilter(ACTION_NETWORK_STATS_POLL);
    mContext.registerReceiver(mPollReceiver, pollFilter, READ_NETWORK_USAGE_HISTORY, mHandler);

    // listen for uid removal to clean stats
    final IntentFilter removedFilter = new IntentFilter(ACTION_UID_REMOVED);
    mContext.registerReceiver(mRemovedReceiver, removedFilter, null, mHandler);

    // listen for user changes to clean stats
    final IntentFilter userFilter = new IntentFilter(ACTION_USER_REMOVED);
    mContext.registerReceiver(mUserReceiver, userFilter, null, mHandler);

    // persist stats during clean shutdown
    final IntentFilter shutdownFilter = new IntentFilter(ACTION_SHUTDOWN);
    mContext.registerReceiver(mShutdownReceiver, shutdownFilter);

    try {
        mNetworkManager.registerObserver(mAlertObserver);
    } catch (RemoteException e) {
        // ignored; service lives in system_server
    }

    registerPollAlarmLocked();
    registerGlobalAlert();
}

NetworkStatsService启动后, 注册了一个定时广播com.android.server.action.NETWORK_STATS_POLL, 每隔一段时间就会定时拉取当前系统消耗的流量统计数据, 收到该广播后, 系统会尝试将统计数据写入到磁盘永久保存下来:


private BroadcastReceiver mPollReceiver = new BroadcastReceiver() {
    @Override
    public void onReceive(Context context, Intent intent) {
        // on background handler thread, and verified UPDATE_DEVICE_STATS
        // permission above.
        performPoll(FLAG_PERSIST_ALL);

        // verify that we're watching global alert
        registerGlobalAlert();
    }
};

// 拉取当前流量数据
private void performPoll(int flags) {
    synchronized (mStatsLock) {
        mWakeLock.acquire();
        try {
            performPollLocked(flags);
        } finally {
            mWakeLock.release();
        }
    }
}

方法performPollLocked首先会获取当前的统计数据快照, 然后将其自动写入到磁盘/data/system/netstats目录:

recordSnapShotLocked()实际通过NetworkManagementService提供的接口从/proc/net/xt_qtaguid这个目录读取当前的历史统计数据并将其保存到mDevRecorder/mXtRecorder/mUidRecorder中
根据传入的标志位, 来确定各个NetworkStatsRecorder是否将数据写入磁盘: 可以强制写入(forcePersistLocked), 也可以等到消耗流量达到阈值(就是之前说的2MB)之后再写入(maybePersistLocked)


private void performPollLocked(int flags) {
    if (!mSystemReady) return;

    final boolean persistNetwork = (flags & FLAG_PERSIST_NETWORK) != 0;
    final boolean persistUid = (flags & FLAG_PERSIST_UID) != 0;
    final boolean persistForce = (flags & FLAG_PERSIST_FORCE) != 0;

    // TODO: consider marking "untrusted" times in historical stats
    final long currentTime = mClock.millis();

    try {
        recordSnapshotLocked(currentTime);
    } catch (IllegalStateException e) {
        Log.wtf(TAG, "problem reading network stats", e);
        return;
    } catch (RemoteException e) {
        // ignored; service lives in system_server
        return;
    }

    // persist any pending data depending on requested flags
    if (persistForce) {
        mDevRecorder.forcePersistLocked(currentTime);
        mXtRecorder.forcePersistLocked(currentTime);
        mUidRecorder.forcePersistLocked(currentTime);
        mUidTagRecorder.forcePersistLocked(currentTime);
    } else {
        if (persistNetwork) {
            mDevRecorder.maybePersistLocked(currentTime);
            mXtRecorder.maybePersistLocked(currentTime);
        }
        if (persistUid) {
            mUidRecorder.maybePersistLocked(currentTime);
            mUidTagRecorder.maybePersistLocked(currentTime);
        }
    }
}

最后, 我们来看看流量统计的数据是如何写入磁盘, 又如何从磁盘读取的. Android将系统消耗的流量按照时间切割成一段段固定时间长度的统计值(NetworkStatsHistory), 并将其与NetworkIdentitySet(表示一个网口集合)组成一个统计的哈希列表(NetworkStatsCollection), 然后每次更新当前消耗的流量时, NetworkStatsRecorder都会不断的将数据写入到磁盘:

NetworkStatsRecorder中包含了两个流量统计数据: 当前未写入磁盘的数据(pending)以及开机以来的统计数据(mSinceBoot)
FileRotator负责将NetworkStatsRecorder中的数据定时写入到磁盘, 并按照一定的老化时间来创建新的统计文件, 而且每个统计文件在达到一定的生命周期后, 会自动被删除

流量统计数据的读写

利用标签来统计特定Socket流量

TrafficStats中提供了接口, 可以在特定的socket(也可以使用socket对应的文件描述符)上打上标签,从而实现对每个应用你内部的流量消耗进行细分.Android系统已经定义了部分的TAG值, 比如用户DHCP协议的数据(TAG_SYSTEM_DHCP), 用于获取NTP网络时间的流量(TAG_SYSTEM_NTP), 用于探测网络的流量(TAG_SYSTEM_PROBE).


public class TrafficStats {

    /**
    * Default tag value for {@link DownloadManager} traffic.
    *
    * @hide
    */
    public static final int TAG_SYSTEM_DOWNLOAD = 0xFFFFFF01;

    /**
    * Default tag value for {@link MediaPlayer} traffic.
    *
    * @hide
    */
    public static final int TAG_SYSTEM_MEDIA = 0xFFFFFF02;

    /**
    * Default tag value for {@link BackupManager} backup traffic; that is,
    * traffic from the device to the storage backend.
    *
    * @hide
    */
    public static final int TAG_SYSTEM_BACKUP = 0xFFFFFF03;

    /**
    * Default tag value for {@link BackupManager} restore traffic; that is,
    * app data retrieved from the storage backend at install time.
    *
    * @hide
    */
    public static final int TAG_SYSTEM_RESTORE = 0xFFFFFF04;

    /**
    * Default tag value for code (typically APKs) downloaded by an app store on
    * behalf of the app, such as updates.
    *
    * @hide
    */
    public static final int TAG_SYSTEM_APP = 0xFFFFFF05;

    /** @hide */
    public static final int TAG_SYSTEM_DHCP = 0xFFFFFF40;
    /** @hide */
    public static final int TAG_SYSTEM_NTP = 0xFFFFFF41;
    /** @hide */
    public static final int TAG_SYSTEM_PROBE = 0xFFFFFF42;
    /** @hide */
    public static final int TAG_SYSTEM_NEIGHBOR = 0xFFFFFF43;
    /** @hide */
    public static final int TAG_SYSTEM_GPS = 0xFFFFFF44;
    /** @hide */
    public static final int TAG_SYSTEM_PAC = 0xFFFFFF45;

    ....

    /**
    * Set active tag to use when accounting {@link Socket} traffic originating
    * from the current thread. Only one active tag per thread is supported.
    * <p>
    * Changes only take effect during subsequent calls to
    * {@link #tagSocket(Socket)}.
    * <p>
    * Tags between {@code 0xFFFFFF00} and {@code 0xFFFFFFFF} are reserved and
    * used internally by system services like {@link DownloadManager} when
    * performing traffic on behalf of an application.
    *
    * @see #clearThreadStatsTag()
    */
    public static void setThreadStatsTag(int tag) {
        NetworkManagementSocketTagger.setThreadSocketStatsTag(tag);
    }

    /**
    * Set active tag to use when accounting {@link Socket} traffic originating
    * from the current thread. Only one active tag per thread is supported.
    * <p>
    * Changes only take effect during subsequent calls to
    * {@link #tagSocket(Socket)}.
    * <p>
    * Tags between {@code 0xFFFFFF00} and {@code 0xFFFFFFFF} are reserved and
    * used internally by system services like {@link DownloadManager} when
    * performing traffic on behalf of an application.
    *
    * @return the current tag for the calling thread, which can be used to
    *         restore any existing values after a nested operation is finished
    */
    public static int getAndSetThreadStatsTag(int tag) {
        return NetworkManagementSocketTagger.setThreadSocketStatsTag(tag);
    }

}

...

    /**
   * Tag the given {@link Socket} with any statistics parameters active for
   * the current thread. Subsequent calls always replace any existing
   * parameters. When finished, call {@link #untagSocket(Socket)} to remove
   * statistics parameters.
   *
   * @see #setThreadStatsTag(int)
   */
  public static void tagSocket(Socket socket) throws SocketException {
      SocketTagger.get().tag(socket);
  }

  /**
   * Remove any statistics parameters from the given {@link Socket}.
   * <p>
   * In Android 8.1 (API level 27) and lower, a socket is automatically
   * untagged when it's sent to another process using binder IPC with a
   * {@code ParcelFileDescriptor} container. In Android 9.0 (API level 28)
   * and higher, the socket tag is kept when the socket is sent to another
   * process using binder IPC. You can mimic the previous behavior by
   * calling {@code untagSocket()} before sending the socket to another
   * process.
   */
  public static void untagSocket(Socket socket) throws SocketException {
      SocketTagger.get().untag(socket);
  }

要使用一个socket的标签其实很简单, 只要在创建通讯的socket的连接后, 主动调用setThreadStatsTag就可以了, 来看一个示例:


private void setupSocket(int sockType, int prot, 
        long writeTimeout, long readTimeout, int destPort) throws ErrnoException, IOException {
    // 将当前socket打上PROBE标签
    int oldTag = TrafficStats.getAndSetThreadStatsTag(TrafficStats.TAG_SYSTEM_PROBE);
    try {
        mFd = Os.socket(mAddrFamily, sockType, prot);
    } finally {
        TrafficStats.setThreadStatsTag(oldTag);
    }

    Os.setsockoptTimeval(mFd, SOL_SOCKET, SO_SNDTIMEO, StructTimeval.fromMillis(writeTimeout));
    Os.setsockoptTimeval(mFd, SOL_SOCKET, SO_RCVTIMEO, StructTimeval.fromMillis(readTimeout));
    if (!TextUtils.isEmpty(mIface)) {
        Os.setsockoptIfreq(mFd, SOL_SOCKET, SO_BINDTODEVICE, mIface);
    }
    Os.connect(mFd, mTarget, destPort);

    mSockAddr = Os.getsockname(mFd);
}

如果要获取某个UID对应的标签数据, 只要调用NetworkStatsManager.java的queryDetailsForUid接口, 传入对应的开始/结束时间就可以了:


/**
 * Query network usage statistics details for a given uid.
 *
 * #see queryDetailsForUidTagState(int, String, long, long, int, int, int)
 */
public NetworkStats queryDetailsForUid(int networkType, String subscriberId,
        long startTime, long endTime, int uid) throws SecurityException {
    return queryDetailsForUidTagState(networkType, subscriberId, startTime, endTime, uid,
        NetworkStats.Bucket.TAG_NONE, NetworkStats.Bucket.STATE_ALL);
}

/**
 * Query network usage statistics details for a given uid and tag.
 *
 * #see queryDetailsForUidTagState(int, String, long, long, int, int, int)
 */
public NetworkStats queryDetailsForUidTag(int networkType, String subscriberId,
        long startTime, long endTime, int uid, int tag) throws SecurityException {
    return queryDetailsForUidTagState(networkType, subscriberId, startTime, endTime, uid,
        tag, NetworkStats.Bucket.STATE_ALL);
}

总结

Android流量统计在9.0还是基于xt_qtaguid来实现的, 后面的版本都会通过BPF来做(参考文章BPF与eBPF).

原文作者：Jason Wang

更新日期：2022-03-16, 12:33:48

Next Post

由新冠病毒想到的一些事情
Previous Post

利用SSH隧道访问局域网

CATALOG

1. Android流量统计实现
2. 利用标签来统计特定Socket流量
3. 总结