Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2174 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 1.0388. Min Range: -1.0388

Max Act: 1.0388. Min Act: -0.0003

Data Index: 1041431 (C4 (Web Text))

Max Activating Token Index: 554

Text #1

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9871. Min Act: -0.0003

Data Index: 409444 (C4 (Web Text))

Max Activating Token Index: 796

Text #2

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9926. Min Act: -0.0003

Data Index: 157196 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9784. Min Act: -0.0003

Data Index: 541228 (C4 (Web Text))

Max Activating Token Index: 187

Text #4

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9433. Min Act: -0.0003

Data Index: 1233599 (C4 (Web Text))

Max Activating Token Index: 130

Text #5

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9702. Min Act: -0.0003

Data Index: 698801 (C4 (Web Text))

Max Activating Token Index: 132

Text #6

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9406. Min Act: -0.0003

Data Index: 129431 (C4 (Web Text))

Max Activating Token Index: 794

Text #7

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9145. Min Act: -0.0003

Data Index: 252230 (C4 (Web Text))

Max Activating Token Index: 445

Text #8

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8403. Min Act: -0.0003

Data Index: 1187363 (C4 (Web Text))

Max Activating Token Index: 575

Text #9

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8514. Min Act: -0.0003

Data Index: 1145140 (C4 (Web Text))

Max Activating Token Index: 47

Text #10

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8258. Min Act: -0.0003

Data Index: 878312 (C4 (Web Text))

Max Activating Token Index: 520

Text #11

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9064. Min Act: -0.0003

Data Index: 887126 (C4 (Web Text))

Max Activating Token Index: 335

Text #12

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.9886. Min Act: -0.0003

Data Index: 1153521 (C4 (Web Text))

Max Activating Token Index: 848

Text #13

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8725. Min Act: -0.0003

Data Index: 1182330 (C4 (Web Text))

Max Activating Token Index: 565

Text #14

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8937. Min Act: -0.0003

Data Index: 952686 (C4 (Web Text))

Max Activating Token Index: 532

Text #15

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8115. Min Act: -0.0003

Data Index: 271721 (C4 (Web Text))

Max Activating Token Index: 292

Text #16

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8483. Min Act: -0.0003

Data Index: 899684 (C4 (Web Text))

Max Activating Token Index: 744

Text #17

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8471. Min Act: -0.0003

Data Index: 502125 (C4 (Web Text))

Max Activating Token Index: 781

Text #18

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8402. Min Act: -0.0003

Data Index: 136710 (C4 (Web Text))

Max Activating Token Index: 470

Text #19

Max Range: 1.0388. Min Range: -1.0388

Max Act: 0.8389. Min Act: -0.0003

Data Index: 1319354 (C4 (Web Text))

Max Activating Token Index: 495