MediaExtractor、MediaMuxer 分离和合成 mp4

MediaExtractor

视音频分离器，将一些格式的视频分离出视频轨道和音频轨道。
主要流程，也是主要的 API：

• setDataSource(String path)：设置数据源
• getTrackCount()：获取通道数
• getTrackFormat(int index)：获取通道格式
• readSampleData(ByteBuffer byteBuf, int offset)：读取指定通道中的数据
• getSampleTime()：获取当前时间戳
• advance()：下一帧
• release()：释放资源

MediaMuxer

视音频合成器，将视频和音频合成相应的格式

• MediaMuxer(String path, int format)：初始化输出文件名称和输出文件的格式：mpeg4
• addTrack(MediaFormat format)：添加轨道，返回 track Index，会在 writeSampleData 中使用
• start()：开始合成文件
• writeSampleData(int, ByteBuffer, MediaCodec.BufferInfo)：写数据
• stop()：停止合成文件
• release()：释放资源

大体流程：

以下是分离 mp4 音频和视频的源码及注释：

private String seperateMedia(String fileName, boolean isAudio) {String type = isAudio ? "audio/" : "video/";MediaExtractor mMediaExtractor = new MediaExtractor();;MediaMuxer mMediaMuxer = null;try {mMediaMuxer = new MediaMuxer(getSDPath() + "/" + fileName, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);// 获取assert中的资源文件AssetFileDescriptor fileDescriptor = getAssetFileSource();// 设置资源文件mMediaExtractor.setDataSource(fileDescriptor.getFileDescriptor(), fileDescriptor.getStartOffset(), fileDescriptor.getLength());int mMediaIndex = 0;for (int i = 0; i < mMediaExtractor.getTrackCount(); i++) {//获取码流的详细格式/配置信息MediaFormat format = mMediaExtractor.getTrackFormat(i);String mine = format.getString(MediaFormat.KEY_MIME);// 查找音频："audio/" 或者视频："video/"的轨道if (mine.startsWith(type)) {mMediaIndex = i;break;}}// 选择感兴趣的轨道mMediaExtractor.selectTrack(mMediaIndex);// 获取通道格式，可以自己新建，但是有坑MediaFormat mediaFormat = mMediaExtractor.getTrackFormat(mMediaIndex);int muxerTrackIndex = mMediaMuxer.addTrack(mediaFormat);// 当采集视频的使用，需要获取帧率，音频轨道没有这个参数int framerate = isAudio ? 0 : mediaFormat.getInteger(MediaFormat.KEY_FRAME_RATE);mMediaMuxer.start();ByteBuffer byteBuffer = ByteBuffer.allocate(500 * 1024);int readSize = 0;// writeSampleData需要BufferInfo参数MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();while((readSize = mMediaExtractor.readSampleData(byteBuffer, 0)) > 0) {bufferInfo.size = readSize;bufferInfo.flags = mMediaExtractor.getSampleFlags(); //设置为关键帧等bufferInfo.offset = 0;if (!isAudio) { // 时间戳，音频和视频的处理方式不一样bufferInfo.presentationTimeUs += 1000 * 1000 / framerate;} else {bufferInfo.presentationTimeUs = mMediaExtractor.getSampleTime();}mMediaMuxer.writeSampleData(muxerTrackIndex, byteBuffer, bufferInfo);Log.d("getSampleTime", "seperateMedia: " + mMediaExtractor.getSampleTime() );mMediaExtractor.advance(); //下一帧}return "success";} catch (IOException e) {e.printStackTrace();} finally {// 释放资源if(mMediaExtractor != null ) {mMediaExtractor.release();mMediaExtractor = null;}if(mMediaMuxer != null) {mMediaMuxer.stop();mMediaMuxer.release();mMediaMuxer = null;}}return null;}

坑点：

bufferInfo 信息中需要写入正确的 presentationTimeUs，而看网上的参考基本都是 MediaExtractor.getSampleTime()，但在实际使用中会异常报错，也有使用通过计算前两个关键帧的差值，然后逐步增加，因为而且 presentationTimeUs 是需要递增的，但是实际上最后分离出来的视频模糊卡顿的现象。

在 Android 中如何提取和生成 mp4 文件 中看到根据帧率来计算 presentationTimeUs，效果比较正常，结合音频使用 mMediaExtractor.getSampleTime();

视频： int frameRate=mediaFormat=getInteger(MediaFormat.KEY_FRAME_RATE); //1s 的帧数
bufferInfo.presentationTimeUs = 1000 * 1000/frameRate; // 对于帧率为 x f/s 的视频而言，时间戳的间隔就是 1000/x ms
音频： bufferInfo.presentationTimeUs = mMediaExtractor.getSampleTime();

最近发现两个问题：
1. 为什么 getSampleTime() 不是逐步递增的？时间戳难道不是应该持续上升的吗？
2. 在某些机型上出现没有 MediaFormat.KEY_FRAME_RATE，该怎么计算 presentationTimeUs？

源码

public class MediaExtractorActivity extends AppCompatActivity implements View.OnClickListener{private static String TAG = MediaExtractorActivity.class.getSimpleName();public static String[] STORAGE = {Manifest.permission.READ_EXTERNAL_STORAGE, Manifest.permission.WRITE_EXTERNAL_STORAGE};@Overrideprotected void onCreate(@Nullable Bundle savedInstanceState) {super.onCreate(savedInstanceState);setContentView(R.layout.activity_mediaextractor);findViewById(R.id.seperate_audio_btn).setOnClickListener(this);findViewById(R.id.seperate_media_btn).setOnClickListener(this);findViewById(R.id.muxer_btn).setOnClickListener(this);}@Overridepublic void onClick(View view) {if(ActivityCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE) != PackageManager.PERMISSION_GRANTED) {ActivityCompat.requestPermissions(this, STORAGE, 2);return;}switch (view.getId()) {case R.id.seperate_audio_btn:new MediaAsyncTask().execute(1);break;case R.id.seperate_media_btn:new MediaAsyncTask().execute(2);break;case R.id.muxer_btn:new MediaAsyncTask().execute(3);break;default:break;}findViewById(R.id.seperate_audio_btn).setEnabled(false);findViewById(R.id.seperate_media_btn).setEnabled(false);findViewById(R.id.muxer_btn).setEnabled(false);}private class MediaAsyncTask extends AsyncTask<Integer, Integer, String> {@Overrideprotected void onPreExecute() {super.onPreExecute();Toast.makeText(MediaExtractorActivity.this, "开始转化", Toast.LENGTH_SHORT).show();}@Overrideprotected String doInBackground(Integer... param) {if (param.length < 1){return null;}switch (param[0]) {case 1:return seperateMedia("audio.mp4", true);case 2:return seperateMedia("video.mp4", false);case 3:return muxerMediaAndAudio("video.mp4","audio.mp4", "result.mp4");default:return null;}}@Overrideprotected void onPostExecute(String s) {findViewById(R.id.seperate_audio_btn).setEnabled(true);findViewById(R.id.seperate_media_btn).setEnabled(true);findViewById(R.id.muxer_btn).setEnabled(true);if (s != null) {Toast.makeText(MediaExtractorActivity.this, "转化完成 " + s, Toast.LENGTH_LONG).show();} else {Toast.makeText(MediaExtractorActivity.this, "转化失败", Toast.LENGTH_SHORT).show();}}}private String muxerMediaAndAudio(String mediaFileName, String audioFileName, String resultName) {File mediaFile = new File(getSDPath(), mediaFileName);File audioFile = new File(getSDPath(), audioFileName);if (!mediaFile.exists() || !audioFile.exists()) {return "音视频文件不存在";}MediaMuxer mMediaMuxer = null;MediaExtractor mMediaExtractor = new MediaExtractor();MediaExtractor mAudioExtractor = new MediaExtractor();try {mMediaMuxer = new MediaMuxer(getSDPath() + "/" + resultName, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);mMediaExtractor.setDataSource(mediaFile.getAbsolutePath());mAudioExtractor.setDataSource(audioFile.getAbsolutePath());int mMediaTrack = 0;int mAudioTrack = 0;for (int i = 0; i < mMediaExtractor.getTrackCount(); i++) {String mine = mMediaExtractor.getTrackFormat(i).getString(MediaFormat.KEY_MIME);if (mine.startsWith("video/")) {mMediaTrack = i;break;}}for (int i = 0; i < mAudioExtractor.getTrackCount(); i++) {String mine = mAudioExtractor.getTrackFormat(i).getString(MediaFormat.KEY_MIME);if (mine.startsWith("audio/")) {mAudioTrack = i;break;}}mMediaExtractor.selectTrack(mMediaTrack);mAudioExtractor.selectTrack(mAudioTrack);MediaFormat mediaFormat = mMediaExtractor.getTrackFormat(mMediaTrack);MediaFormat audioFormat = mAudioExtractor.getTrackFormat(mAudioTrack);MediaCodec.BufferInfo mediaBufferInfo = new MediaCodec.BufferInfo();MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();int mediaMuxerTrack = mMediaMuxer.addTrack(mediaFormat);int audioMuxerTrack = mMediaMuxer.addTrack(audioFormat);mMediaMuxer.start();ByteBuffer byteBuffer = ByteBuffer.allocate(500 * 1024);int readSize = 0;int mMediaFramerate = mediaFormat.getInteger(MediaFormat.KEY_FRAME_RATE);while ((readSize = mMediaExtractor.readSampleData(byteBuffer, 0)) > 0) {mediaBufferInfo.presentationTimeUs += 1000 * 1000 / mMediaFramerate;mediaBufferInfo.offset = 0;mediaBufferInfo.flags = mMediaExtractor.getSampleFlags();mediaBufferInfo.size = readSize;mMediaMuxer.writeSampleData(mediaMuxerTrack, byteBuffer, mediaBufferInfo);mMediaExtractor.advance();}while ((readSize = mAudioExtractor.readSampleData(byteBuffer, 0)) > 0) {audioBufferInfo.presentationTimeUs = mAudioExtractor.getSampleTime();audioBufferInfo.offset = 0;audioBufferInfo.flags = mAudioExtractor.getSampleFlags();audioBufferInfo.size = readSize;mMediaMuxer.writeSampleData(audioMuxerTrack, byteBuffer, audioBufferInfo);mAudioExtractor.advance();}mMediaMuxer.stop();return getSDPath() + resultName;} catch (IOException e) {e.printStackTrace();} finally {if(mMediaExtractor != null ) {mMediaExtractor.release();mMediaExtractor = null;}if(mAudioExtractor != null ) {mAudioExtractor.release();mAudioExtractor = null;}if(mMediaMuxer != null) {mMediaMuxer.release();mMediaMuxer = null;}}return null;}private String seperateMedia(String fileName, boolean isAudio) {String type = isAudio ? "audio/" : "video/";MediaExtractor mMediaExtractor = new MediaExtractor();;MediaMuxer mMediaMuxer = null;try {mMediaMuxer = new MediaMuxer(getSDPath() + "/" + fileName, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);// 获取assert中的资源文件AssetFileDescriptor fileDescriptor = getAssetFileSource();// 设置资源文件mMediaExtractor.setDataSource(fileDescriptor.getFileDescriptor(), fileDescriptor.getStartOffset(), fileDescriptor.getLength());int mMediaIndex = 0;for (int i = 0; i < mMediaExtractor.getTrackCount(); i++) {//获取码流的详细格式/配置信息MediaFormat format = mMediaExtractor.getTrackFormat(i);String mine = format.getString(MediaFormat.KEY_MIME);// 查找音频："audio/" 或者视频："video/"的轨道if (mine.startsWith(type)) {mMediaIndex = i;break;}}// 选择感兴趣的轨道mMediaExtractor.selectTrack(mMediaIndex);// 获取通道格式，可以自己新建，但是有坑MediaFormat mediaFormat = mMediaExtractor.getTrackFormat(mMediaIndex);int muxerTrackIndex = mMediaMuxer.addTrack(mediaFormat);// 当采集视频的使用，需要获取帧率，音频轨道没有这个参数int framerate = 0;if (mediaFormat.containsKey(MediaFormat.KEY_FRAME_RATE)) {framerate = mediaFormat.getInteger(MediaFormat.KEY_FRAME_RATE);}mMediaMuxer.start();ByteBuffer byteBuffer = ByteBuffer.allocate(500 * 1024);int readSize = 0;// writeSampleData需要BufferInfo参数MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();while((readSize = mMediaExtractor.readSampleData(byteBuffer, 0)) > 0) {bufferInfo.size = readSize;bufferInfo.flags = mMediaExtractor.getSampleFlags(); //设置为关键帧等bufferInfo.offset = 0;if (framerate != 0) { // 时间戳，音频和视频的处理方式不一样bufferInfo.presentationTimeUs += 1000 * 1000 / framerate;} else {bufferInfo.presentationTimeUs = mMediaExtractor.getSampleTime();}mMediaMuxer.writeSampleData(muxerTrackIndex, byteBuffer, bufferInfo);Log.d("getSampleTime", "seperateMedia: " + mMediaExtractor.getSampleTime() );mMediaExtractor.advance(); //下一帧}mMediaMuxer.stop();return "success";} catch (IOException e) {e.printStackTrace();} finally {// 释放资源if(mMediaExtractor != null ) {mMediaExtractor.release();mMediaExtractor = null;}if(mMediaMuxer != null) {mMediaMuxer.release();mMediaMuxer = null;}}return null;}private AssetFileDescriptor getAssetFileSource() throws IOException {AssetManager manager = getAssets();return manager.openFd("screen_recorder.mp4");}public static String getSDPath() {// 判断是否挂载if (Environment.getExternalStorageState().equals(Environment.MEDIA_MOUNTED)) {return Environment.getExternalStorageDirectory().getAbsolutePath();}return Environment.getRootDirectory().getAbsolutePath();} }

参考：
http://blog.51cto.com/ticktick/1710743

总结

以上是生活随笔为你收集整理的MediaExtractor、MediaMuxer 分离和合成 mp4的全部内容，希望文章能够帮你解决所遇到的问题。

如果觉得生活随笔网站内容还不错，欢迎将生活随笔推荐给好友。