Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3091 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.7967. Min Act: -0.0000

Data Index: 1403347 (Python Code)

Max Activating Token Index: 817

Text #1

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.7515. Min Act: -0.0000

Data Index: 905732 (C4 (Web Text))

Max Activating Token Index: 756

Text #2

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.4311. Min Act: -0.0000

Data Index: 1521898 (Python Code)

Max Activating Token Index: 903

Text #3

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.5399. Min Act: -0.0000

Data Index: 1474033 (Python Code)

Max Activating Token Index: 396

Text #4

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.4825. Min Act: -0.0001

Data Index: 1219761 (C4 (Web Text))

Max Activating Token Index: 410

Text #5

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.4320. Min Act: -0.0000

Data Index: 226437 (C4 (Web Text))

Max Activating Token Index: 806

Text #6

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.5458. Min Act: -0.0000

Data Index: 1079698 (C4 (Web Text))

Max Activating Token Index: 420

Text #7

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.4175. Min Act: -0.0000

Data Index: 1531646 (Python Code)

Max Activating Token Index: 315

Text #8

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.5015. Min Act: -0.0001

Data Index: 1288409 (C4 (Web Text))

Max Activating Token Index: 557

Text #9

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.3832. Min Act: -0.0000

Data Index: 642064 (C4 (Web Text))

Max Activating Token Index: 156

Text #10

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.4190. Min Act: -0.0001

Data Index: 1138944 (C4 (Web Text))

Max Activating Token Index: 63

Text #11

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.5219. Min Act: -0.0000

Data Index: 1025937 (C4 (Web Text))

Max Activating Token Index: 866

Text #12

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.4690. Min Act: -0.0000

Data Index: 104482 (C4 (Web Text))

Max Activating Token Index: 538

Text #13

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.4023. Min Act: -0.0000

Data Index: 161304 (C4 (Web Text))

Max Activating Token Index: 44

Text #14

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.3470. Min Act: -0.0000

Data Index: 859124 (C4 (Web Text))

Max Activating Token Index: 151

Text #15

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.2968. Min Act: -0.0000

Data Index: 1117336 (C4 (Web Text))

Max Activating Token Index: 45

Text #16

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.3256. Min Act: -0.0000

Data Index: 1436624 (Python Code)

Max Activating Token Index: 492

Text #17

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.2444. Min Act: -0.0000

Data Index: 1529687 (Python Code)

Max Activating Token Index: 6

Text #18

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.3153. Min Act: -0.0001

Data Index: 1237610 (C4 (Web Text))

Max Activating Token Index: 459

Text #19

Max Range: 1.7967. Min Range: -1.7967

Max Act: 1.2362. Min Act: -0.0000

Data Index: 1446761 (Python Code)

Max Activating Token Index: 889