|Year : 2021 | Volume
| Issue : 1 | Page : 146-154
Network Pharmacology Evaluation Method Guidance - Draft
Institute for TCM-X, MOE Key Laboratory of Bioinformatics/Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing 100084, P. R. China
|Date of Submission||26-Jan-2020|
|Date of Acceptance||14-Feb-2021|
|Date of Web Publication||01-Mar-2021|
Prof. Shao Li
Institute for TCM-X, MOE Key Laboratory of Bioinformatics/Bioinformatics Division, BNRIST, Department of Automation, Tsinghua University, Beijing 100084
P. R. China
Source of Support: None, Conflict of Interest: None
|How to cite this article:|
Li S. Network Pharmacology Evaluation Method Guidance - Draft. World J Tradit Chin Med 2021;7:146-54
Note: Some of the document contents may involve certain patents, the identification of which is not the responsibility of the institution that releases the document.
Main drafting organizations: Tsinghua University, Specialty Committee of Network Pharmacology of World Federation of Chinese Medicine Societies (WFCMS).
Drafting organizations participated: Institute of Chinese Medicine, Chinese Academy of Chinese Medical Sciences, The Fifth Medical Center of Chinese PLA General Hospital, Beijing University of Chinese Medicine, Shanghai University of Traditional Chinese Medicine, Anhui University of Chinese Medicine, Tianjin University of Traditional Chinese Medicine, Zhejiang Chinese Medical University, University of Macau, Shenyang Pharmaceutical University, Wenzhou Medical University, Zunyi Medical University, Lanzhou University, Beijing Jiaotong University, Tianjin International Joint Academy of Biomedicine, and Tsinghua-Fuzhou Data Technology Research Center.
The main drafter: Shao Li.
Other drafter participated (listed in the alphabetic order of the surname): Yi-Tao Chen, Qing-Yang Ding, Jian-Ye Dai, Xian-Chun Duan, Yuan-Jia Hu (from Macau, China), Xin-Xing Lai, Qing-Fei Liu, Ming Niu, Rong-Wu Xiang, Zheng Xiang, Hai-Yu Xu, Xin Wang, Hong-Jun Yang, Kuo Yang, Ming Yang, Hai-Yang Yu, Bo Zhang, Peng Zhang, Qian-Ru Zhang, Hui-Hui Zhao, and Jing Zhao.
This document was drafted in accordance with the WFCMS Regulations for International Organization Standard and the Working Regulation for Formulation and Publication of Standard (SCM 0001-2009), both issued by the WFCMS.
This document is issued by the WFCMS, and all copyrights are reserved to WFCMS.
| Introduction|| |
Network pharmacology is an interdisciplinary discipline newly developed in the systematic research of drugs based on artificial intelligence and Big Data. It stresses the holistic system-level and biological networks when analyzing the molecular association laws between drugs and treatment objects. Focusing on studying drugs, it provides new ideas, especially for Chinese medicine research, which is based on a complex system. It is also widely applied to explore the active compounds of drugs and Chinese medicine, explain the overall action mechanisms, and analyze the compatibility regularity of drug pairs and formulas. It has provided new scientific and technological support for the rational clinical use of drugs and drug development.
With an increasing influence and application in the Big Data era, network pharmacology now faces great development opportunities and challenges in terms of theoretical analyses, algorithm development, and applications. The issue of integrating massive clinical and experimental data and combining scientific verification to reveal the regulation mechanisms of network pharmacology to carry out its research more effectively has become the main concern of researchers. In addition, there are multiple problems in the current network pharmacology studies, such as uneven research quality, lack of data standardization, and insufficient scientific verification. Establishing a rigorous, scientific, and unified standard for evaluating network pharmacology studies is urgently required to ensure this emerging discipline's healthy development.
This guidance establishes the association mechanism between drugs and diseases in biomolecular networks based on the “network target,” the main theory of network pharmacology, and forms a “network target-system regulation”-based research mode and method that provides a new way to understand and explain the interactions between drugs and biological systems.
By establishing the normative evaluation standard of network pharmacology, this document aims to make the “network target-system regulation”-based research mode, a new generation of drug research paradigm which is more rigorous and scientific and is widely recognized and promotes the standardized application of network pharmacology in drug analysis and experiment, and clinical pharmacological studies, thus promoting the rapid, healthy, and orderly development of the discipline.
Due to the diversity and complexity of network pharmacology research, and the limitation on the technical competence and research conditions of the current related studies, there are still many problems in its evaluation method and content that needs to be further resolved. Therefore, this document will be continuously revised following the development of network pharmacology research.
| Guideline for Network Pharmacology Evaluation Method|| |
This document specifies the data collection and network analysis in the process of network pharmacology research, as well as experiment verification principles, procedures, and evaluation indicators.
This document applies to analyzing the effects of botanical, animal, chemical, and biological drugs and disease mechanisms for researchers or reviewers engaged in network pharmacology and related disciplines.
2 Normative references
The following referenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.
- GB/T 36344 Information Technology – Evaluation Indicators for Data Quality
- GB/T 34945 Information Technology – Data Provenance Descriptive Model
- GB/T 6379 Accuracy (Trueness and Precision) of Measurement Methods and Results
- AIOSS-01 Artificial Intelligence – Assessment Specification for Deep Learning Algorithms
- GA/T 1587 Test Specifications for Automatic Speaker Recognition System.
3 Terms and definitions
For the purpose of this document, the following terms and definitions apply.
3.1 Network pharmacology
Network pharmacology integrates systems biology, bioinformatics, network science, and other disciplines and analyzes the molecular relationship between drugs and treatment objects to reveal the systematic pharmacological mechanism of drugs from an overall perspective of the system level and biological network, thereby guiding the drug development and clinical diagnosis and treatment. It is a new original discipline of systematic drug research in the era of artificial intelligence and Big Data.
3.2 Network targets
The “network target” is the key network points in the biomolecular network referring to the mechanism-based associations between drugs and diseases that quantitatively indicate the overall regulation mechanisms of drugs, including the key molecules, pathways, or modules.
3.3 Material basis
The chemical components and their related properties of drugs including the name, type, physical and chemical properties, and pharmacokinetic parameters of these chemical substances.
3.4 Drug targets
The biomolecules inside the body that can be affected by drugs including protein, RNA, and DNA.
3.5 Network analysis
The deep analysis and mining of the biomolecular networks related to diseases or drug interventions include basic network analyses (degree, betweenness, clustering coefficient, and other basic network parameters) and deep network analyses (network modules and dynamic analysis).
3.6 Biological functional annotation
This refers to the annotation and enrichment of biological functions such as the pathway and gene ontology in disease-related biomolecular networks or those with drug interventions using bioinformatic methods.
3.7 Biological functional prediction
Based on the regularity results of network analyses, biological functions are predicted with bioinformatics and artificial intelligence.
This refers to the ratio between the amount of information correctly detected and the total number of related information in the retrieval system.
This refers to the ratio between the amount of information detected correctly by verification and the total amount of information detected.
(Source: GA/T 1587, 3.13. Some contents were modified.)
This refers to the ratio between the amount of information correctly detected and the total amount of relevant information in the retrieval system. It is a measure of the ability of an information retrieval system to detect relevant information.
(Source: GA/T 1587, 3.14. Some contents were modified.)
Note: The recall is also called sensitivity.
This refers to the ability of an analysis system to correctly distinguish or detect the information when there are interfering substances.
3.12 F- Measure
The weighted mean of precision and recall is used to evaluate the effectiveness of the analysis methods. In addition, the equation of F-measure needs to be modified, as follows:
F = (α+ 1) × Precision × Recall/α× (Precision + Recall)
When α = 1, it refers to the F1 value.
(Source: GA/T 1587, 3.15. Some contents are modified.)
This refers to the consistency and stability degree of the results obtained through repeated analyses of the same object with the same method under certain conditions.
This refers to the degree of a corresponding analysis method or means that can accurately judge the analyzed objects.
3.15 Coincidence rate
This refers to the rate of consistent results obtained by repeated analyses of the same object with the same method under the same conditions.
4 Evaluation requirements
The network pharmacology evaluation is conducted from three aspects – reliability, standardization, and rationality. The specific requirements are listed as follows:
- Evaluation of reliability: This refers to whether the access to the main data and their correlation information, design of the software algorithm and analysis method, and selection and model building of the verification method are reliable and meet the analysis requirements. For selecting evaluation elements and indicators, please see [Table A.1] in Annex A.
- Evaluation of standardization: This evaluates whether the processes such as data information extraction and conversion, software/algorithm development, network construction and analysis, and the experimental verification are standard, as well as whether the application of relevant techniques is accurate to ensure the accuracy and reproducibility of the analysis results. For the evaluation elements and indicators, please see [Table A.2] in Annex A.
- Evaluation of rationality: This evaluates whether the data screening and filtering, network analysis index selection and threshold value determination, validation model, and test index selection are rational. For evaluation elements and indicators, please see [Table A.3] in Annex A.
5 Evaluation contents
Note: The network pharmacology evaluation process includes general evaluation and scalability evaluation. The specific contents are illustrated in [Table 1].
5.1 General evaluation
Network pharmacology research requires the evaluation of various analysis objects (disease and drugs) and analysis purposes (database development, algorithm development, mechanism research, diagnosis, and treatment discovery, drug development, etc.) to ensure the authenticity of the analysis results.
5.2 Scalability evaluation
This refers to certain evaluations conducted more deeply and objectively to evaluate different analysis objects and analysis purposes to improve the reliability of the network pharmacology analysis results.
6 Technical contents of evaluations
Note: The technical contents of network pharmacology evaluations can be divided into data collection, network analyses, and result verification, as illustrated in [Table 2].
6.1 Evaluation of reliability
6.1.1 Data collection
This mainly evaluates the reliability of the data sources and collection method for the drug material basis, biological targets, and other basic data. The evaluation contents are as follows:
- Data source: This evaluates the sources of data used, such as literature, databases, and experimental data
- Data information: This evaluates the general information of data used, such as the total amount of data, collection time, and versions of relevant databases
- Associated information: This evaluates the relevant contents of different data, such as the amount of associated information and the type.
Data verified by rigorous experiments or from authoritative literature are preferable. For those from software prediction and databases with nonliterature sources, their reliability evaluation should be provided as much as possible, and those with high reliability are recommended.
6.1.2 Network analysis
In biological target network analyses, the reliability of a newly developed algorithm or the correctness and stability of the selected analysis method is evaluated. The main evaluation content is as follows:
- Software and algorithm: This evaluates the software or algorithm applied during the analysis, such as the well-applied or original software or algorithm
- Analysis method: This evaluates the specific methods used during the analysis, such as the overall technical routes and experimental ideas.
When applying the newly developed algorithm, the evaluation indicators commonly used include the implementation of algorithms, the correctness of algorithm function, and the algorithm performance, for example, the accuracy, recall rate, and F-score.
When applying mature analytical methods in related fields to network pharmacology analyses, standard data sets or experimental data are recommended for reliability analyses on methods to be used. The former requires data collection consultations when evaluating their reliability.
When the network analysis method has been used for research, the source of the analysis method should be clearly marked.
6.1.3 Result verification
The reliability and repeatability of the methods adopted should be mainly evaluated to ensure the credibility of the verification results and final conclusions. The main evaluation contents are as follows:
- Verification methods: These evaluate the specific methods adopted during the result verification, such as literature verifications, computer-assisted verifications, experimental verifications, and clinical trials
- Model construction: This evaluates the model construction methods and end point indicators adopted during the result verification. Among them, the methods of the evaluation model include computer model construction, in vivo and in vitro experiment model construction, and enrollment of clinical research cohort, while that of end point indicators include golden indicators and representative indicators.
In vivo and in vitro experiments or clinical trials are preferred for verification.
6.2 Evaluation of standardization
6.2.1 Data collection
The completeness of data information, the clarity of data extraction, and the standardization of data processing in the process of data collection are evaluated. The main evaluation contents are as follows:
- Information extraction: This evaluates the methods of extracting or obtaining the relevant information on the contents to be analyzed, such as the rule and scope of information extraction
- Information conversion: This evaluates the conversion methods of different data information, such as the information conversion of different databases and abnormal information processing.
6.2.2 Network analysis
The definiteness of the evaluation process, the standardization of the evaluation method, and the traceability of the analysis methods are evaluated. The main evaluation contents are as follows:
- Algorithm implementation: This evaluates the implementation process of the analysis software or algorithm to be adopted, such as the call mode of software or algorithm and the development method of the original algorithm
- Analysis path: This evaluates the main process of analysis, such as the process of network analysis.
6.2.3 Result verification
When verifying the analysis results of network pharmacology, the standardization of operational procedures, results analysis, and other verification processes are evaluated, including computer simulation processes, in vivo and in vitro experimental processes, and the process of clinical trials.
6.3 Evaluation of rationality
6.3.1 Data collection
When evaluating data collection, the rationality of the information transmission route, information processing, and other content is evaluated. The main evaluation content is as follows:
- Data provenance: This evaluates the basis of the data to be analyzed, such as the specific information of references and specific results of experimental or clinical trials
- Data filtering: This evaluates the relevant process of selecting the data to be analyzed, such as the screening principles and screening conditions.
For screening pharmacodynamic components, analysis objects, absorption pathways, effective parts, markedly effective components, metabolic forms, bioavailability, druggability, and other factors affecting pharmacodynamic behaviors should be fully considered. Parameters affecting pharmacodynamic behaviors should be selected for screening pharmacodynamic components.
For screening targets and their interactions, data credibility and acquisition method rationality should be particularly investigated.
6.3.2 Network analysis
The rationality of analysis indicators used to screen important targets (clusters), key pharmacodynamic components (clusters), and other factors is evaluated, including software/algorithm parameters and network analysis parameters. Among them, relevant indexes of network analyses include the characteristics of nodes and edges (such as node degree, node centrality, and edge weight), network cohesion (such as density, clustering coefficients, subgraphs, and connectivity), network separability (such as hierarchical clustering, and spectrum segmentation), network modularity, and network motifs.
For the specific diseases studied, the above indicators should be comprehensively considered to establish a reasonable disease syndrome biomolecular network and determine appropriate network targets.
If a new computing method is applied in determining the pharmacodynamic component clusters and network targets, the importance of the selected pharmacodynamic components and network targets to the overall network stability should also be investigated.
6.3.3 Result verification
This evaluates the rationality of trials designed for verifying network pharmacology analysis results, detection methods, and other specific implementation processes. The main evaluation contents are as follows:
- Validation model: This evaluates the model used for result validation, such as computer models, experimental models, and clinical research cohorts
- Detection index: This evaluates the specific detection index referred to in result verification, such as output results of computer simulations, detection indexes of algorithm efficiency, and detection indexes of experiments or clinical trials.
For the newly developed analysis algorithms of network pharmacology, suitable performance evaluation indicators must be provided according to the result type, and the prediction results need to be verified. Standard data sets can be applied for verification if necessary.
7 Evaluation process
The general process of network pharmacology evaluation is illustrated in [Figure 1].
Evaluation Elements and Indicators of Network Pharmacology
| A.1 The Elements and Indicators for Reliability Evaluation|| |
For reliability evaluation elements and indicators, please see Table A.1.
A.2 The Elements and Indicators for Standardization Evaluation
For standardization evaluation elements and indicators, please see Table A.2.
A.3 The Elements and Indicators for Rationality Evaluation
For rationality evaluation elements and indicators, please see Table A.3.
Case of a Research Report on Network Pharmacology
| B.1 Title and Abstract|| |
Both the title and abstract must indicate the research object. The abstract must also briefly describe the research objective, method, result, conclusion, and significance of the network pharmacology analysis.
This should elaborately elucidate the research background, principles, and other contents. For the specific research objective, the exact clinical efficacy or experimental evidence should be provided as much as possible, such as the real-world research result, high-quality randomized controlled trial (RCT) study, cohort study, or reliable research data.
| B.3 Methods and Results|| |
B.3.1 Research design
An overview of the key content of the research design must be reported, such as the research object, method and type (such as dry experiment and combination of dry and wet experiments), and data source (the way of obtaining data). A research flowchart must be provided when necessary.
B.3.2 Research object
B.3.2.1 For disease-related research, the standard name of the disease, and if necessary, the current version of the ICD disease code number must be provided. Studies on traditional Chinese medicine syndrome must write its name correctly, which must meet the relevant requirements of the National Standard for Clinical Diagnosis and Treatment Terminology of Traditional Chinese Medicine (syndrome part).
B.3.2.2 For drug-related research, the standard names of compounds or the unique identified information of the compounds, and if necessary, their structural formulas must be provided. For the compound formula of Chinese herbs, the source and composition of the formula, as well as the standard names of every Chinese herb it includes, must be provided. Latin names are recommended for Chinese herbs.
B.3.3 Data source
B.3.3.1 This must describe the data source as elaborately as possible, such as the literature data, database retrieval, or experimental data. For the ways of obtaining data, all repeatable detailed parameters, and the description of data source limitations, if necessary, must be provided.
B.3.3.2 For data obtained by database retrieval, the database name, source, version, and retrieval data and strategy, as well as the database references for web databases, must be provided. The descriptive statistics, such as the access address and data collection, must also be provided if necessary.
For data obtained from the literature, the original references must be provided, and how these reference data are obtained must be described. If necessary, descriptive statistics must be carried out on the literature data.
For data obtained from experiments, specific experimental methods, objects (related to this research), and conditions, as well as the way of obtaining experimental results, must be provided. If necessary, the verification results of the experimental methods must also be provided.
B.3.4 Network analysis
B.3.4.1 This must elaborately describe the relevant elements and their interactions in network construction and the methods and indicators of network analysis. The biological significance must be provided in the network analysis result. In the network construction, the relationship between the relevant network elements and the overall data must be described. For a filtered subset, its rationality must be reflected; specific screening methods, basis, and principles must be provided; and its relationship with the research objective must be elucidated.
B.3.4.2 The network analysis method must specifically describe the following:
- Analysis content and indicators, such as the basic parameters of the network (the characteristics of nodes and edges, the cohesive characteristics of the network, and the modularity of the network), and their relationship with the research objective
- Network analysis strategy. If necessary, the pathway flowchart of the analysis must be provided. For a nonoriginal network analysis method, the source of the method, its analysis parameters, references, and name and version of the analysis software must be provided. If necessary, the sensitivity analysis results must also be provided for the analysis algorithm with parameter selection
- For the original analysis algorithm, the principles, specific implementation steps, implementation methods, parameter settings, and programming language of the new algorithm must be described. If a standard data set is used, its reliability analysis result must be provided. If necessary, the results of comparisons with popular algorithms and robustness analyses must also be provided
- The network analysis results must have clear biological significance. If there is a biological function annotation, the type of the annotation (such as gene ontology, pathway, and disease) and the method, parameters, software name, version of the annotation, and their relationship with the research objective must be elaborately described.
B.3.5 Result verification
B.3.5.1 This must describe the verification method, strategy, and its relationship with the research objective in detail. Verification with combined multiple methods is recommended. For nonoriginal methods, verification with only computer-aided methods or literature data is not recommended
B.3.5.2 This must be provided for result verification:
- For computer-aided verification, the name, reason, source, reference, parameter setting, software name and version, and comparison analysis result of the algorithm must be provided. While for literature data verification, the basis, literature source, and retrieval data must be provided, and the ways of obtaining data from the original literature and their reliability analysis result must be described
- For experimental research verification, the experimental objects, materials, models, methods, the sample collection and processing method, detection indicators, data acquisition method, and analysis results must be elaborately described. Its relationship with the research objective must also be clarified
- For clinical research verification, the research design and objects, inclusion and exclusion criteria, informed consent, trial implementation process, data management, sample collection strategy and method, sample processing method, detection indicators, and statistical analysis results must be elaborately described. The relationship with the research objective must be clarified. If necessary, approval documents from the ethics committee must be provided.
B.3.5.3 For studies focusing on developing network pharmacology algorithms, databases, and computational analysis platforms, it is necessary to verify the prediction analysis results obtained by the algorithm, database, and platform.
- According to the type of predictive result, the result evaluated by the indicators of the performance analysis and reliability evaluation must be provided. The consistency evaluation against different studies on a standard data set is also recommended.
- The results reliability is verified by rigorous clinical trials or experimental studies. Among them, the reliability verification based on clinic and experiment has a high level of evidence.
| B.4 Discussion|| |
The overall result explanation must be given prudently according to the research objective. On this basis, the research must increase the analysis on the generalizability of the method, while the applied research must increase the analysis on the interpretability of the research results. In addition, the limitations of research results must be analyzed, including the source of uncertainty and any potential factors that may affect the research result.
| B.5 Other Content|| |
In addition to the above content, whether the original data are accessible in the research report must be clarified, and if necessary, access to it must be provided. For the analysis method, the software packages used in the research and their version numbers and if necessary the analysis algorithm source code must be provided. For other original data, methods, analysis results, other supplementary information not mentioned in the article, and access to the supplementary information must be provided.
[Table 1], [Table 2]