Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4733 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5926. Min Range: -2.5926

Max Act: 2.5926. Min Act: -0.0000

Data Index: 867172 (C4 (Web Text))

Max Activating Token Index: 42

Text #1

Max Range: 2.5926. Min Range: -2.5926

Max Act: 2.4507. Min Act: 0.0000

Data Index: 629050 (C4 (Web Text))

Max Activating Token Index: 396

Text #2

Max Range: 2.5926. Min Range: -2.5926

Max Act: 2.0307. Min Act: -0.0000

Data Index: 727239 (C4 (Web Text))

Max Activating Token Index: 550

Text #3

Max Range: 2.5926. Min Range: -2.5926

Max Act: 2.0154. Min Act: -0.0000

Data Index: 1304842 (C4 (Web Text))

Max Activating Token Index: 856

Text #4

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.8751. Min Act: -0.0000

Data Index: 969877 (C4 (Web Text))

Max Activating Token Index: 987

Text #5

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.8095. Min Act: -0.0000

Data Index: 870811 (C4 (Web Text))

Max Activating Token Index: 632

Text #6

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.8761. Min Act: -0.0000

Data Index: 1042516 (C4 (Web Text))

Max Activating Token Index: 925

Text #7

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.6758. Min Act: -0.0000

Data Index: 842861 (C4 (Web Text))

Max Activating Token Index: 987

Text #8

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.7461. Min Act: -0.0000

Data Index: 861377 (C4 (Web Text))

Max Activating Token Index: 481

Text #9

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.7115. Min Act: -0.0000

Data Index: 619468 (C4 (Web Text))

Max Activating Token Index: 593

Text #10

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.7038. Min Act: -0.0000

Data Index: 12981 (C4 (Web Text))

Max Activating Token Index: 443

Text #11

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.6650. Min Act: -0.0000

Data Index: 580547 (C4 (Web Text))

Max Activating Token Index: 465

Text #12

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.6556. Min Act: -0.0000

Data Index: 1046939 (C4 (Web Text))

Max Activating Token Index: 705

Text #13

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.6359. Min Act: -0.0000

Data Index: 1202150 (C4 (Web Text))

Max Activating Token Index: 325

Text #14

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.6922. Min Act: -0.0000

Data Index: 508267 (C4 (Web Text))

Max Activating Token Index: 923

Text #15

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.5354. Min Act: -0.0000

Data Index: 573430 (C4 (Web Text))

Max Activating Token Index: 969

Text #16

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.5231. Min Act: 0.0000

Data Index: 1045614 (C4 (Web Text))

Max Activating Token Index: 273

Text #17

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.4837. Min Act: -0.0000

Data Index: 903444 (C4 (Web Text))

Max Activating Token Index: 527

Text #18

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.4279. Min Act: -0.0000

Data Index: 796716 (C4 (Web Text))

Max Activating Token Index: 982

Text #19

Max Range: 2.5926. Min Range: -2.5926

Max Act: 1.4809. Min Act: -0.0000

Data Index: 577026 (C4 (Web Text))

Max Activating Token Index: 1005