Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2890 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1064. Min Range: -1.1064

Max Act: 1.1064. Min Act: -0.0000

Data Index: 1545342 (Python Code)

Max Activating Token Index: 116

Text #1

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8909. Min Act: -0.0000

Data Index: 1384325 (Python Code)

Max Activating Token Index: 858

Text #2

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.9270. Min Act: -0.0000

Data Index: 1508646 (Python Code)

Max Activating Token Index: 905

Text #3

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.9691. Min Act: -0.0000

Data Index: 1510769 (Python Code)

Max Activating Token Index: 252

Text #4

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8730. Min Act: -0.0000

Data Index: 1617083 (Python Code)

Max Activating Token Index: 556

Text #5

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8463. Min Act: -0.0000

Data Index: 1477854 (Python Code)

Max Activating Token Index: 892

Text #6

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8900. Min Act: -0.0000

Data Index: 1588149 (Python Code)

Max Activating Token Index: 371

Text #7

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8399. Min Act: -0.0000

Data Index: 212742 (C4 (Web Text))

Max Activating Token Index: 125

Text #8

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8351. Min Act: -0.0000

Data Index: 835288 (C4 (Web Text))

Max Activating Token Index: 468

Text #9

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8724. Min Act: -0.0000

Data Index: 1361728 (Python Code)

Max Activating Token Index: 318

Text #10

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8099. Min Act: -0.0000

Data Index: 1523760 (Python Code)

Max Activating Token Index: 889

Text #11

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8848. Min Act: -0.0000

Data Index: 1454595 (Python Code)

Max Activating Token Index: 396

Text #12

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8558. Min Act: -0.0000

Data Index: 1603926 (Python Code)

Max Activating Token Index: 432

Text #13

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8283. Min Act: -0.0000

Data Index: 1608465 (Python Code)

Max Activating Token Index: 939

Text #14

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.8168. Min Act: -0.0000

Data Index: 1407113 (Python Code)

Max Activating Token Index: 356

Text #15

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.7682. Min Act: -0.0000

Data Index: 1563056 (Python Code)

Max Activating Token Index: 426

Text #16

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.7798. Min Act: -0.0000

Data Index: 1489688 (Python Code)

Max Activating Token Index: 621

Text #17

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.7307. Min Act: -0.0000

Data Index: 1542221 (Python Code)

Max Activating Token Index: 396

Text #18

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.7294. Min Act: -0.0000

Data Index: 316844 (C4 (Web Text))

Max Activating Token Index: 573

Text #19

Max Range: 1.1064. Min Range: -1.1064

Max Act: 0.7723. Min Act: -0.0000

Data Index: 1401323 (Python Code)

Max Activating Token Index: 675