MySQL事務(wù)RUNNING狀態(tài)引發(fā)的Transactiontimedout:deadline問題_MySQL

前言：
朋友說簡(jiǎn)單的查詢導(dǎo)致Transaction timed out: deadline問題，懷疑是數(shù)據(jù)庫(kù)表鎖了，

1，應(yīng)用故障描述Deadline問題：
--- The error occurred in META-INF/ibatis/ITEM_sqlmap.xml.
--- The error occurred while executing query.
--- Check the SELECT I.ID, I.SHOP_ID, I.ITEM_GROUP_ID, I.ITEM_GROUP_NAME, I.ITEM_NAME, I.LIST_PRICE, I.PL_PRICE, I.PROTECTION_PRICE, I.MALL, I.STAT, I.LOGISTICS, I.TYPE, I.ITEM_CATEGORY_ID, I.BRAND_ID, I.BRAND, I.FAVOUR_NUM, I.IS_SUBSCRIBE, I.VOLUME, I.WEIGHT, I.INVENTORY, I.RELEASE_DATE, I.OFF_REASON, I.IS_DEL, I.CREATED_DATE, I.UPDATED_DATE, I.SIMPLE_DESCRIPTION, I.VIRTUAL_BEGIN_DATE, I.VIRTUAL_END_DATE, I.SEQ_NUM, IPC.PICTURE_PATH FROM ITEM AS I INNER JOIN ITEM_PICTURE AS IPC ON I.ITEM_GROUP_ID = IPC.ITEM_GROUP_ID where I.ID = ? AND IPC.TYPE='1' AND IPC.IS_DEL='0' .
--- Check the SQL Statement (preparation failed).
--- Cause: org.springframework.transaction.TransactionTimedOutException: Transaction timed out: deadline was Tue Nov 25 13:33:07 CST 2014
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:204)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryForObject(MappedStatement.java:120)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:518)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(SqlMapExecutorDelegate.java:493)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(SqlMapSessionImpl.java:106)
at org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(SqlMapClientTemplate.java:273)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:203)
... 41 more
Caused by: org.springframework.transaction.TransactionTimedOutException: Transaction timed out: deadline was Tue Nov 25 13:33:07 CST 2014
at org.springframework.transaction.support.ResourceHolderSupport.checkTransactionTimeout(ResourceHolderSupport.java:141)
at org.springframework.transaction.support.ResourceHolderSupport.getTimeToLiveInMillis(ResourceHolderSupport.java:130)
at org.springframework.transaction.support.ResourceHolderSupport.getTimeToLiveInSeconds(ResourceHolderSupport.java:114)
at org.springframework.jdbc.datasource.DataSourceUtils.applyTimeout(DataSourceUtils.java:275)
at org.springframework.jdbc.datasource.DataSourceUtils.applyTransactionTimeout(DataSourceUtils.java:257)
at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:244)
at com.sun.proxy.$Proxy324.prepareStatement(Unknown Source)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.prepareStatement(SqlExecutor.java:497)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:175)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.sqlExecuteQuery(MappedStatement.java:221)
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeQueryWithCallback(MappedStatement.java:189)
... 47 more

2，檢查Innodb存儲(chǔ)引擎狀態(tài)以及表鎖狀態(tài)
SHOW ENINGE INNODB STATUS；沒有死鎖信息以及其它異常信息；去查詢系統(tǒng)表INNODB_LOCKS、INNODB_LOCK_WAITS表都為NULL，只有INNODB_TRX表有記錄，并且處于長(zhǎng)時(shí)間RUNNING狀態(tài)，判斷是因?yàn)槭聞?wù)沒有提交或者回滾的緣故。
mysql> SELECT * FROM `INNODB_TRX`;
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | trx_operation_state | trx_tables_in_use | trx_tables_locked | trx_lock_structs | trx_lock_memory_bytes | trx_rows_locked | trx_rows_modified | trx_concurrency_tickets | trx_isolation_level | trx_unique_checks | trx_foreign_key_checks | trx_last_foreign_key_error | trx_adaptive_hash_latched | trx_adaptive_hash_timeout | trx_is_read_only | trx_autocommit_non_locking |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| 19183390 | RUNNING | 2014-11-25 15:39:30 | NULL | NULL | 9 | 940341 | NULL | NULL | 0 | 0 | 5 | 1248 | 3 | 4 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 9762 | 0 | 0 |
| 19183153 | RUNNING | 2014-11-25 15:36:41 | NULL | NULL | 0 | 940206 | NULL | NULL | 0 | 0 | 0 | 376 | 0 | 0 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 9411 | 0 | 0 |
| 19183139 | RUNNING | 2014-11-25 15:36:28 | NULL | NULL | 0 | 940238 | NULL | NULL | 0 | 0 | 0 | 376 | 0 | 0 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 9937 | 0 | 0 |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
3 rows in set (0.00 sec)

mysql> SELECT * FROM `INNODB_LOCKS`;
Empty set (0.00 sec)

mysql> SELECT * FROM `INNODB_LOCK_WAITS`;
Empty set (0.00 sec)

mysql>

仔細(xì)check從中可以看出，沒有表鎖，沒有行鎖，沒有鎖等待，只有事務(wù)RUNNING沒有提交或者回滾。臨時(shí)解決辦法，kill掉這些事務(wù)所在的線程。

3，問題重現(xiàn)，查看事務(wù)表記錄
mysql> SELECT * FROM `INNODB_TRX`;
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | trx_operation_state | trx_tables_in_use | trx_tables_locked | trx_lock_structs | trx_lock_memory_bytes | trx_rows_locked | trx_rows_modified | trx_concurrency_tickets | trx_isolation_level | trx_unique_checks | trx_foreign_key_checks | trx_last_foreign_key_error | trx_adaptive_hash_latched | trx_adaptive_hash_timeout | trx_is_read_only | trx_autocommit_non_locking |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| 19196180 | RUNNING | 2014-11-25 17:41:10 | NULL | NULL | 12 | 942663 | NULL | NULL | 0 | 0 | 4 | 1248 | 2 | 8 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 9810 | 0 | 0 |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
1 rows in set (0.00 sec)

mysql>

4，去Slow log和binlog里面分析
去slow log里面看942663線程ID的slow記錄，沒有找到， 去看binlog里面942663線程ID的DML記錄，有如下2條記錄：
#141125 17:41:10 server id 230 end_log_pos 118147 CRC32 0x6f2402a1 Query thread_id=942663 exec_time=0 error_code=0
SET TIMESTAMP=1416908470/*!*/;
BEGIN
/*!*/;
# at 118147
#141125 17:41:10 server id 230 end_log_pos 118231 CRC32 0x0219bed2 Table_map: `business_db`.`SHOP_CASH_COUPON_USER_REF` mapped to number 178
# at 118231
#141125 17:41:10 server id 230 end_log_pos 118298 CRC32 0xc6665994 Write_rows: table id 178 flags: STMT_END_F
### INSERT INTO `business_db`.`SHOP_CASH_COUPON_USER_REF`
### SET
### @1=4859
### @2=284
### @3=2425
### @4='0'
### @5='2014-11-25 17:41:10'
### @6=NULL
# at 118298
#141125 17:41:10 server id 230 end_log_pos 118411 CRC32 0x93f6d105 Table_map: `business_db`.`CASH_TICKET` mapped to number 727
# at 118411
#141125 17:41:10 server id 230 end_log_pos 118703 CRC32 0xe4b314ad Update_rows: table id 727 flags: STMT_END_F
### UPDATE `business_db`.`CASH_TICKET`
### WHERE
### @1=19956
### @2=1416886592
### @3=NULL
### @4=NULL
### @5=1
### @6='2014-12-31 00:00:00'
### @7='2014-11-25 00:00:00'
### @8=NULL
### @9=NULL
### @10=NULL
### @11=NULL
### @12=NULL
### @13=NULL
### @14=NULL
### @15=NULL
### @16='5c2483b3033b30c6b948d6a971c87f1d'
### @17='CASH-TICKET-1'
### @18='0'
### @19=000000050.000000000
### @20=NULL
### @21=284
### @22='0'
### SET
### @1=19956
### @2=1416908470
### @3='PL.1qaz2wsx'
### @4=NULL
### @5=1
### @6='2014-12-31 00:00:00'
### @7='2014-11-25 00:00:00'
### @8=NULL
### @9=2425
### @10='PL.1qaz2wsx'
### @11='1qaz2wsx@163.com'
### @12='4859'
### @13=NULL
### @14=NULL
### @15=NULL
### @16='5c2483b3033b30c6b948d6a971c87f1d'
### @17='CASH-TICKET-1'
### @18='0'
### @19=000000050.000000000
### @20=1416908470
### @21=284
### @22='0'
# at 118703
#141125 17:41:10 server id 230 end_log_pos 118734 CRC32 0x6949012e Xid = 16199116
COMMIT/*!*/;

看到這個(gè)執(zhí)行成功的DML操作和一直RUNNING的事務(wù)時(shí)間上比較吻合，所以通過這binlog的INSERT語(yǔ)句和UPDATE語(yǔ)句，找到應(yīng)用的一個(gè)業(yè)務(wù)模塊的方法，發(fā)現(xiàn)其異常處理模塊沒有及時(shí)COMMIT和ROLLBACK的。
添加上ROLLBACK處理后，再測(cè)試N次，沒有出現(xiàn)過報(bào)錯(cuò)信息，而執(zhí)行SELECT * FROM `INNODB_TRX`;也沒有記錄，這表示事務(wù)都及時(shí)COMMIT或者ROLLBACK了。

5，期間遇到的額外問題
Cause: java.sql.SQLException: Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.; nested exception is com.ibatis.common.jdbc.exception.NestedSQLException:
--- The error occurred while applying a parameter map.
--- Check the ITEM.updateByInventory-InlineParameterMap.
--- Check the statement (update failed).
--- Cause: java.sql.SQLException: Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.
處理方法：將BINLOG_FORMAT設(shè)置成MIXED即可，SET GLOBAL BINLOG_FORMAT = MIXED;

6，總結(jié)
這個(gè)問題看似解決了，但是可能還有更多的細(xì)節(jié)沒有梳理，為什么事務(wù)開啟后沒有ROLLBACK或者COMMIT，后續(xù)執(zhí)行SELECT就會(huì)報(bào)錯(cuò)呢？自己想來是DML形成排它鎖X，而查詢是有共享鎖S，X和S是互斥的（關(guān)于Innodb鎖請(qǐng)參考：http://blog.itpub.net/26230597/viewspace-1315111/），所以就出問題了，至于源碼層的底層分析，有待以后繼續(xù)深究，如果有遇到此類問題的朋友，歡迎share下你的處理思路以及分析過程，謝謝。

----------------------------------------------------------------------------------------------------------------
<版權(quán)所有，文章允許轉(zhuǎn)載，但必須以鏈接方式注明源地址，否則追究法律責(zé)任!>
原博客地址： http://blog.itpub.net/26230597/viewspace-1346680/
原作者：黃杉 (mchdba)
----------------------------------------------------------------------------------------------------------------

聲明：本網(wǎng)頁(yè)內(nèi)容旨在傳播知識(shí)，若有侵權(quán)等問題請(qǐng)及時(shí)與本網(wǎng)聯(lián)系，我們將在第一時(shí)間刪除處理。TEL:177 7030 7066 E-MAIL:11247931@qq.com