计算化学公社

标题: ChatGPT貌似无法对GIF图片进行识别 [打印本页]

作者
Author: 扣扣 时间: 2023-5-10 11:10
标题: ChatGPT貌似无法对GIF图片进行识别
本帖最后由扣扣于 2023-5-10 20:12 编辑

此前有听说过ChatGPT对于图片的识别和生成（灌水杂谈？），今天偶（wu）然（liao）想起来测试一下New Bing （GPT-4）对图片的识别和描述能力（GPT-3.5没联网能力，所以没法让它听我的...）。为了使GPT能成功读取我想要让它读取的图片数据，我只能使用网络上面已有的图片（鄙人并没有什么多余的网络空间来储存图片，除了qq空间

），想着是如果他能够识别出关键信息，那么后续再考虑传图片到网络上让它读取。

首先，从Google Earth Engine找了一张GIF动图（https://developers. google. com/static/ earth-engine/images/datasets/gifs/nasa_nex_07_2018.gif）。描述的是2018年7月全球的温度变化动图。使用如下描述语句进行输入：

Help me describe the figure on the following website: https://developers. google. com/static/ earth-engine/images/datasets/gifs/nasa_nex_07_2018.gif.

GPT的结果为：
The figure on the website is an animated GIF that shows the monthly average surface air temperature for the year 20181. The data is derived from the NASA Earth Exchange Global Daily Downscaled Projections (NEX-GDDP) dataset, which is a collection of climate model outputs that are downscaled to a 25 km resolution using a statistical technique1. The figure is an example of how Google Earth Engine can be used to visualize and analyze geospatial data at a planetary scale23. Google Earth Engine is a platform that combines a multi-petabyte catalog of satellite imagery and geospatial datasets with cloud computing capabilities and geospatial APIs23. It enables scientists, researchers, and developers to perform geospatial processing at scale and create dynamic applications for Earth science data23.

应该是从网站上（nasa_nex_07_2018）的信息识别出来了这个数据集是来自NASA Earth Exchange Global Daily Downscaled Projections (NEX-GDDP) dataset，从（developers.google.com/static/earth-engine/images/datasets）识别出来了这个图是从Google Earth Engine上扒下来的，进而描述了这些网站上关于这个图片的一些标签，但并没有包含更多的地理信息（无法做一些类似卷积的特征抽取）。

鄙人继续使用如下语句进行提问：
Tell me more details about the temperature in China from this GIF.

GPT的回复为：
The GIF shows that the temperature in China varied from month to month in 2018, with the highest temperatures occurring in July and August and the lowest temperatures occurring in January and December1. The temperature anomalies were generally positive, meaning that the temperatures were above the long-term average for most regions1. According to the National Centers for Environmental Information (NCEI), the 2018 temperature for Asia was 1.26°C (2.27°F) above average and the seventh highest in the 109-year record2. China experienced a heat wave in summer 2018, especially in northern China, where the maximum, mean, and minimum temperatures all reached record highs3. The heat wave was caused by the intensified and more northward western Pacific subtropical high, weakened Northeast China cold vortex, and positive geopotential height anomaly from northern China to the Sea of Japan3. On the other hand, China also experienced extremely low temperatures in winter 2018, especially in Mohe, the northernmost city of China, where the temperature plunged to minus 53°C (minus 63.4 degrees Fahrenheit) on December 6, 2018, which was its coldest ever recorded4. The low temperature was attributed to a strong cold air outbreak that swept across northeastern China4. These extreme temperature events in China are consistent with the projected trends of climate change, which indicate that China will face substantially hotter summers and winters by 2050 due to global warming.

可见，GPT的回复虽然洋洋洒洒一大堆，但是并没有描述出这个GIF的更多细节，而且最重要的是，所有的描述均来自网络搜索，且忽略了这个GIF只包含了2018年7月信息的事实，所以在当前语句下应该是无法实现对GIF图片的识别能力的

。大概是因为GIF文件？
随后使用相同语句对NASA Earthdata的jpg文件（https://lpdaac. usgs. gov/media/images/MCD12C1_V6_2016_global_hero.original_pQTEM81.jpg）进行了测试，GPT的结果如下：

The figure is a map of the global land cover types derived from the Terra and Aqua combined Moderate Resolution Imaging Spectroradiometer (MODIS) Land Cover Climate Modeling Grid (CMG) (MCD12C1) Version 6 data product1. It shows the dominant land cover class for each 0.05 degree pixel based on the International Geosphere-Biosphere Programme (IGBP) classification scheme1. The IGBP scheme identifies 17 land cover classes, which include 11 natural vegetation classes, 3 developed and mosaicked land classes, and 3 non-vegetated land classes2. The map uses different colors to represent each land cover class, such as green for forests, brown for grasslands, yellow for croplands, and white for snow and ice3.

According to the figure, the land cover types in China vary greatly across different regions and show a zonal pattern of change1. The dominant land cover types in China are barren land, real steppe, desert steppe and forest1. Barren land mainly occurs in the high-altitude areas of the Tibetan Plateau and the western part of the Mongolian Plateau1. Real steppe and desert steppe are widely distributed in the Inner Mongolia Autonomous Region and the northern part of China1. Forests are mainly concentrated in the northeast, southwest and southeast of China1. Other land cover types include meadow steppe, sand, desert, ice, water, cropland and built area1. Cropland and built area are mostly located in the lowland plains and basins along the eastern coast of China, where most of the population and arable land are based2.

还是针对网址标签进行检索得到的结果，这样根本没办法做特征抽取嘛

作者
Author: 扣扣 时间: 2023-5-10 20:08
测试了jpg文件，效果依旧不行，所有的信息都是通过网址内的标签检索得到的，可能是：1）执行语句有问题

；2:目前new bing不能识别图片内的特征信息

作者
Author: wzkchem5 时间: 2023-5-16 18:22

扣扣发表于 2023-5-10 13:08
测试了jpg文件，效果依旧不行，所有的信息都是通过网址内的标签检索得到的，可能是：1）执行语句有问题 ...

可能只是因为从标签搜索更容易？好比有人问我一道我花一个小时也未必做得出来的题，我可能会在网上搜原题，看有没有现成的答案

作者
Author: 扣扣 时间: 2023-5-19 11:03

wzkchem5 发表于 2023-5-16 18:22
可能只是因为从标签搜索更容易？好比有人问我一道我花一个小时也未必做得出来的题，我可能会在网上搜原题 ...

有道理，可能我设置的describe被它理解为只要是描述这张图片的信息了？我再参考参考别的语句试试看

欢迎光临计算化学公社 (http://bbs.keinsci.com/)