Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6023 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.9819. Min Act: -0.0000

Data Index: 61118 (C4 (Web Text))

Max Activating Token Index: 18

Text #1

Max Range: 2.0577. Min Range: -2.0577

Max Act: 2.0577. Min Act: -0.0000

Data Index: 558935 (C4 (Web Text))

Max Activating Token Index: 611

Text #2

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.7846. Min Act: -0.0000

Data Index: 1064014 (C4 (Web Text))

Max Activating Token Index: 68

Text #3

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.6428. Min Act: -0.0000

Data Index: 274239 (C4 (Web Text))

Max Activating Token Index: 751

Text #4

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.5809. Min Act: -0.0000

Data Index: 1235368 (C4 (Web Text))

Max Activating Token Index: 304

Text #5

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.5076. Min Act: -0.0000

Data Index: 1339596 (C4 (Web Text))

Max Activating Token Index: 392

Text #6

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.5435. Min Act: -0.0000

Data Index: 290438 (C4 (Web Text))

Max Activating Token Index: 425

Text #7

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.5562. Min Act: -0.0000

Data Index: 1641576 (Python Code)

Max Activating Token Index: 977

Text #8

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.5415. Min Act: -0.0000

Data Index: 737425 (C4 (Web Text))

Max Activating Token Index: 983

Text #9

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.5434. Min Act: -0.0000

Data Index: 482083 (C4 (Web Text))

Max Activating Token Index: 328

Text #10

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4569. Min Act: -0.0000

Data Index: 298004 (C4 (Web Text))

Max Activating Token Index: 83

Text #11

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4973. Min Act: -0.0000

Data Index: 775894 (C4 (Web Text))

Max Activating Token Index: 638

Text #12

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4536. Min Act: -0.0000

Data Index: 146744 (C4 (Web Text))

Max Activating Token Index: 160

Text #13

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4511. Min Act: -0.0000

Data Index: 809536 (C4 (Web Text))

Max Activating Token Index: 220

Text #14

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.6145. Min Act: -0.0000

Data Index: 937812 (C4 (Web Text))

Max Activating Token Index: 986

Text #15

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4796. Min Act: -0.0000

Data Index: 1117721 (C4 (Web Text))

Max Activating Token Index: 16

Text #16

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4802. Min Act: -0.0000

Data Index: 150118 (C4 (Web Text))

Max Activating Token Index: 449

Text #17

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.3250. Min Act: -0.0000

Data Index: 1182979 (C4 (Web Text))

Max Activating Token Index: 847

Text #18

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4782. Min Act: -0.0000

Data Index: 319221 (C4 (Web Text))

Max Activating Token Index: 719

Text #19

Max Range: 2.0577. Min Range: -2.0577

Max Act: 1.4692. Min Act: -0.0000

Data Index: 789705 (C4 (Web Text))

Max Activating Token Index: 443