Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4421 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3094. Min Range: -2.3094

Max Act: 2.3094. Min Act: -0.0000

Data Index: 76181 (C4 (Web Text))

Max Activating Token Index: 977

Text #1

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.8332. Min Act: -0.0000

Data Index: 1406736 (Python Code)

Max Activating Token Index: 257

Text #2

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.7284. Min Act: -0.0000

Data Index: 1627852 (Python Code)

Max Activating Token Index: 338

Text #3

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.8136. Min Act: -0.0000

Data Index: 1405463 (Python Code)

Max Activating Token Index: 349

Text #4

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.6778. Min Act: -0.0000

Data Index: 1495986 (Python Code)

Max Activating Token Index: 127

Text #5

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.6509. Min Act: -0.0000

Data Index: 1518661 (Python Code)

Max Activating Token Index: 640

Text #6

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.4670. Min Act: 0.0000

Data Index: 1604347 (Python Code)

Max Activating Token Index: 987

Text #7

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.5447. Min Act: 0.0000

Data Index: 1484430 (Python Code)

Max Activating Token Index: 246

Text #8

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.4961. Min Act: -0.0000

Data Index: 1379080 (Python Code)

Max Activating Token Index: 742

Text #9

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.4588. Min Act: -0.0000

Data Index: 1414064 (Python Code)

Max Activating Token Index: 502

Text #10

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.4282. Min Act: -0.0000

Data Index: 1360462 (Python Code)

Max Activating Token Index: 880

Text #11

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.3793. Min Act: -0.0000

Data Index: 1646878 (Python Code)

Max Activating Token Index: 389

Text #12

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.3706. Min Act: -0.0000

Data Index: 1488749 (Python Code)

Max Activating Token Index: 391

Text #13

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.3612. Min Act: 0.0000

Data Index: 1480192 (Python Code)

Max Activating Token Index: 853

Text #14

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.2511. Min Act: -0.0000

Data Index: 1472467 (Python Code)

Max Activating Token Index: 97

Text #15

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.2271. Min Act: -0.0000

Data Index: 1575609 (Python Code)

Max Activating Token Index: 107

Text #16

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.2921. Min Act: -0.0000

Data Index: 1638235 (Python Code)

Max Activating Token Index: 993

Text #17

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.2331. Min Act: 0.0000

Data Index: 1381848 (Python Code)

Max Activating Token Index: 202

Text #18

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.2743. Min Act: -0.0000

Data Index: 1627704 (Python Code)

Max Activating Token Index: 510

Text #19

Max Range: 2.3094. Min Range: -2.3094

Max Act: 1.2368. Min Act: -0.0000

Data Index: 1552210 (Python Code)

Max Activating Token Index: 536