Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2921 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1877. Min Act: -0.0000

Data Index: 111346 (C4 (Web Text))

Max Activating Token Index: 881

Text #1

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1595. Min Act: -0.0000

Data Index: 1055592 (C4 (Web Text))

Max Activating Token Index: 815

Text #2

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1522. Min Act: -0.0000

Data Index: 789086 (C4 (Web Text))

Max Activating Token Index: 630

Text #3

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1475. Min Act: -0.0000

Data Index: 1082219 (C4 (Web Text))

Max Activating Token Index: 565

Text #4

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1628. Min Act: -0.0000

Data Index: 349500 (C4 (Web Text))

Max Activating Token Index: 769

Text #5

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1497. Min Act: -0.0000

Data Index: 291495 (C4 (Web Text))

Max Activating Token Index: 968

Text #6

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1509. Min Act: -0.0000

Data Index: 691990 (C4 (Web Text))

Max Activating Token Index: 927

Text #7

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1450. Min Act: -0.0000

Data Index: 317810 (C4 (Web Text))

Max Activating Token Index: 897

Text #8

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1446. Min Act: -0.0000

Data Index: 415914 (C4 (Web Text))

Max Activating Token Index: 982

Text #9

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1416. Min Act: -0.0000

Data Index: 925794 (C4 (Web Text))

Max Activating Token Index: 688

Text #10

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1489. Min Act: -0.0000

Data Index: 168850 (C4 (Web Text))

Max Activating Token Index: 84

Text #11

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1449. Min Act: -0.0000

Data Index: 21131 (C4 (Web Text))

Max Activating Token Index: 478

Text #12

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1434. Min Act: -0.0000

Data Index: 724458 (C4 (Web Text))

Max Activating Token Index: 993

Text #13

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1428. Min Act: -0.0000

Data Index: 552585 (C4 (Web Text))

Max Activating Token Index: 680

Text #14

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1330. Min Act: -0.0000

Data Index: 953299 (C4 (Web Text))

Max Activating Token Index: 549

Text #15

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1473. Min Act: -0.0000

Data Index: 1038604 (C4 (Web Text))

Max Activating Token Index: 483

Text #16

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1510. Min Act: -0.0000

Data Index: 368671 (C4 (Web Text))

Max Activating Token Index: 922

Text #17

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1284. Min Act: -0.0000

Data Index: 414503 (C4 (Web Text))

Max Activating Token Index: 996

Text #18

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1386. Min Act: -0.0000

Data Index: 1028549 (C4 (Web Text))

Max Activating Token Index: 366

Text #19

Max Range: 0.1877. Min Range: -0.1877

Max Act: 0.1490. Min Act: -0.0000

Data Index: 74845 (C4 (Web Text))

Max Activating Token Index: 885