Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2346 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.2469. Min Act: -0.0000

Data Index: 1229948 (C4 (Web Text))

Max Activating Token Index: 136

Text #1

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.0870. Min Act: -0.0000

Data Index: 639442 (C4 (Web Text))

Max Activating Token Index: 130

Text #2

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.3604. Min Act: -0.0000

Data Index: 1213230 (C4 (Web Text))

Max Activating Token Index: 194

Text #3

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.2334. Min Act: -0.0000

Data Index: 544605 (C4 (Web Text))

Max Activating Token Index: 151

Text #4

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.0458. Min Act: -0.0000

Data Index: 1311934 (C4 (Web Text))

Max Activating Token Index: 97

Text #5

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.2770. Min Act: -0.0000

Data Index: 361417 (C4 (Web Text))

Max Activating Token Index: 61

Text #6

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.8593. Min Act: -0.0000

Data Index: 338336 (C4 (Web Text))

Max Activating Token Index: 118

Text #7

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.1772. Min Act: -0.0000

Data Index: 827290 (C4 (Web Text))

Max Activating Token Index: 107

Text #8

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.5643. Min Act: -0.0000

Data Index: 565852 (C4 (Web Text))

Max Activating Token Index: 738

Text #9

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.6611. Min Act: -0.0000

Data Index: 680860 (C4 (Web Text))

Max Activating Token Index: 956

Text #10

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.0773. Min Act: -0.0000

Data Index: 32509 (C4 (Web Text))

Max Activating Token Index: 120

Text #11

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.4465. Min Act: -0.0000

Data Index: 1004148 (C4 (Web Text))

Max Activating Token Index: 166

Text #12

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.6282. Min Act: -0.0000

Data Index: 1321083 (C4 (Web Text))

Max Activating Token Index: 188

Text #13

Max Range: 2.3604. Min Range: -2.3604

Max Act: 2.0313. Min Act: -0.0000

Data Index: 78773 (C4 (Web Text))

Max Activating Token Index: 830

Text #14

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.5878. Min Act: -0.0000

Data Index: 405152 (C4 (Web Text))

Max Activating Token Index: 720

Text #15

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.4977. Min Act: -0.0000

Data Index: 320957 (C4 (Web Text))

Max Activating Token Index: 825

Text #16

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.4717. Min Act: -0.0000

Data Index: 349333 (C4 (Web Text))

Max Activating Token Index: 915

Text #17

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.4584. Min Act: -0.0000

Data Index: 974438 (C4 (Web Text))

Max Activating Token Index: 149

Text #18

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.7155. Min Act: -0.0000

Data Index: 920856 (C4 (Web Text))

Max Activating Token Index: 916

Text #19

Max Range: 2.3604. Min Range: -2.3604

Max Act: 1.5691. Min Act: -0.0000

Data Index: 37979 (C4 (Web Text))

Max Activating Token Index: 977