Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 933 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.2351. Min Act: -0.0001

Data Index: 78186 (C4 (Web Text))

Max Activating Token Index: 955

Text #1

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.2207. Min Act: -0.0001

Data Index: 139749 (C4 (Web Text))

Max Activating Token Index: 250

Text #2

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.2187. Min Act: -0.0001

Data Index: 329200 (C4 (Web Text))

Max Activating Token Index: 601

Text #3

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.2046. Min Act: -0.0001

Data Index: 100429 (C4 (Web Text))

Max Activating Token Index: 53

Text #4

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1905. Min Act: -0.0001

Data Index: 528180 (C4 (Web Text))

Max Activating Token Index: 773

Text #5

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1899. Min Act: -0.0001

Data Index: 464034 (C4 (Web Text))

Max Activating Token Index: 287

Text #6

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1945. Min Act: -0.0001

Data Index: 821460 (C4 (Web Text))

Max Activating Token Index: 54

Text #7

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.2026. Min Act: -0.0001

Data Index: 1299277 (C4 (Web Text))

Max Activating Token Index: 858

Text #8

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1867. Min Act: -0.0001

Data Index: 876444 (C4 (Web Text))

Max Activating Token Index: 442

Text #9

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1841. Min Act: -0.0001

Data Index: 1138746 (C4 (Web Text))

Max Activating Token Index: 965

Text #10

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1946. Min Act: -0.0001

Data Index: 149027 (C4 (Web Text))

Max Activating Token Index: 103

Text #11

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1777. Min Act: -0.0001

Data Index: 906371 (C4 (Web Text))

Max Activating Token Index: 757

Text #12

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1759. Min Act: -0.0001

Data Index: 491289 (C4 (Web Text))

Max Activating Token Index: 274

Text #13

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1733. Min Act: -0.0001

Data Index: 908443 (C4 (Web Text))

Max Activating Token Index: 321

Text #14

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1805. Min Act: -0.0001

Data Index: 489243 (C4 (Web Text))

Max Activating Token Index: 78

Text #15

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1657. Min Act: -0.0001

Data Index: 1049361 (C4 (Web Text))

Max Activating Token Index: 966

Text #16

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1655. Min Act: -0.0001

Data Index: 843295 (C4 (Web Text))

Max Activating Token Index: 600

Text #17

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1688. Min Act: -0.0001

Data Index: 944479 (C4 (Web Text))

Max Activating Token Index: 256

Text #18

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1361. Min Act: -0.0001

Data Index: 505895 (C4 (Web Text))

Max Activating Token Index: 877

Text #19

Max Range: 0.2351. Min Range: -0.2351

Max Act: 0.1674. Min Act: -0.0001

Data Index: 799115 (C4 (Web Text))

Max Activating Token Index: 404